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We have developed and tested an advanced EVA communications and computing system 
to increase astronaut self-reliance and safety, reducing dependence on continuous 
monitoring and advising from mission control on Earth. This system, called Mobile Agents 
(MA), is voice controlled and provides information verbally to the astronauts through 
programs called “personal agents.” The system partly automates the role of CapCom in 
Apollo — including monitoring and managing EVA navigation, scheduling, equipment 
deployment, telemetry, health tracking, and scientific data collection. EVA data are stored 
automatically in a shared database in the habitat/vehicle and mirrored to a site accessible by 
a remote science team. The program has been developed iteratively in the context of use, 
including six years of ethnographic observation of field geology. Our approach is to develop 
automation that supports the human work practices, allowing people to do what they do 
well, and to work in ways they are most familiar. Field experiments in Utah have enabled 
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empirically discovering requirements and testing alternative technologies and protocols. 
This paper reports on the 2004 system configuration, experiments, and results, in which an 
EVA robotic assistant (ERA) followed geologists approximately 150 m through a winding, 
narrow canyon. On voice command, the ERA took photographs and panoramas and was 
directed to move and wait in various locations to serve as a relay on the wireless network. 
The MA system is applicable to many space work situations that involve creating and 
navigating from maps (including configuring equipment for local topology), interacting with 
piloted and unpiloted rovers, adapting to environmental conditions, and remote team 
collaboration involving people and robots. 


I. Introduction 

E XTRAVEHICULAR ACTIVITIES (EVAs) in space or on land are presently expensive, risky, and highly 
dependent on large teams of ground operation support. The “Mobile Agents” (MA) system is designed to 
automate the role of CapCom in Apollo — including monitoring and managing EVA navigation, scheduling, 
equipment deployment, telemetry, health tracking, and data collection. Mobile Agents uses a multi-agent, distributed 
architecture, 1 ' 7 integrating diverse components in a voice-controlled EVA communications and astronaut assistant 
system, currently including: biosensors, cameras, GPS, EVA Robotic Assistant, and science database. This wireless 
computing and software system proactively transmits data, assisting communication between EVA crew, 
autonomous robots, vehicle or surface habitat crew, and remote support teams. In conventional information 
processing terms, MA provides a voice-commanded, workflow tool that proactively coordinates the actions of 
robotic devices and people, while facilitating strategic and scientific collaboration in human teams. 

Agents — incorporating models of EVA activities and robots/devices represented in a modular, reconfigurable, 
reusable way — store, forward, interpret, and transform data as they become available to help people and a robotic 
network cooperate to make operations more safe, affordable, and effective (e.g., agents could monitor EVA progress 
and consumables to dynamically replan EVAs). A spoken dialogue interface enables people to communicate with 
agents, including robotic controllers. A science database in the habitat is mirrored to Earth, for access and annotation 
by scientists and mission support for collaborative review and planning. 

The Mobile Agent Architecture (MAA) has been developed iteratively in the context of use, including six years 
of ethnographic observation 9 ' 11 and field experiments, 1213 with scientists doing authentic exploration. The design 
was inspired by analysis of Apollo lunar traverses, 14,15 as well as a two-week simulated Mars mission at the Mars 
Desert Research Station (MDRS). 16 The MDRS field “testbed” facility /site provides a test bed for identifying 
requirements, competitively testing alternative technologies/protocols, and training astronauts. 17 

Using the Mobile Agents approach could reduce human monitoring during an EVA from dozens of full-time 
controllers to an alert-based system that contacts and involves people only during anomalous situations (e.g., habitat 
loudspeaker broadcast and email to Earth). Target metrics relative to Apollo include: 100% reduction in manual 
telemetry readouts, 100% transmission of science data to Earth during EVAs, 80% reduction in human monitoring, 
and 60% automation of dynamic EVA replanning. The current system is ready for advanced application. However, 
replicating human CapCom capabilities in navigation interpretation and advising, EVA scheduling, and safety 
monitoring requires development of models with uncertain, judgmental inferences. Such capabilities can be 
gradually incorporated, but in the limit, expert-level human reasoning lies in the realm of open “artificial 
intelligence” research. By following the incremental cycle of analysis, design, implementation, and field 
experimentation, the Mobile Agents system provides EVA computing and communication tools that are always 
useful, with progressively improving capabilities. 

The MAA is applicable to many space work situations. For example, construction involves building and 
navigating maps (including localization and configuring equipment for terrain topology), interacting with piloted 
and unpiloted rovers, changing environmental conditions, and remote team collaboration. 

Subsequent subsections of this introduction provide an overview of objectives and accomplishments, a summary 
of the analog mission cycle, and the scenarios and schedule for the field experiments. Subsequent sections of this 
article describe Field Participants and the Mobile Agents 2004 Configuration, followed by three sections describing 
Crew Collaboration with the Support Team, Crew/RST Collaboration Tools, and Reporting during the Simulated 
Mission. More detailed sections are based on reports written in the field: Pedestrian EVA Scenario (Pooh 's Corner), 
Head of Canyon Scenario Planning, and Long Relay EVA Scenario (Lith Canyon ). Finally, the article provides two 
summary sections: Results and Discussion (including design recommendations) and Conclusions. 
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A. Overview of 2004 Objectives and Accomplishments 

Crew 29 returned to the Mars Desert Research Station (MDRS habitat, “the hab”), five miles from the minuscule 
town of Hanksville, tucked in the corner of the southeast Utah canyons and painted desert, for two weeks of 
experimental operation of the Mobile Agents system. In April 2003, we had shown as Crew 16 how to use MDRS as 
its name suggests — a research station — a protected workplace for scientists and engineers to configure, deploy, and 
test sophisticated technologies and protocols in an extreme environment. 14 In 2004 we connected a database to the 
network in the hab (“ScienceOrganizer”) and made a mirrored web site accessible to a remote science team (RST), 
with a five-minute time delay to simulate interplanetary communication. The RST receives data from the field as it 
is collected during the EVA, plus they provide feedback on the crew’s plans for each day. 

Six people ate and slept in the hab (Clancey, Sierhuis, Alena, Dowding, Garry, and Semple) while the remaining 
group of 12 or so found accommodation in Hanksville. During the day, the habitat was shared by the full team, with 
the Ames and Johnson Space Center (JSC) computer scientists and SUNY-Buffalo geologists sharing the upper deck 
of the habitat. (The JSC team also operated out of a rental truck “office” parked outside the habitat.) Only the EVAs 
were conducted in simulation mode (with two people in suits, and a support crew off camera). We used 
collaboration tools to prepare EVAs, preparing diagrams, outlined notes, plans, and videos that could be shared with 
the RST; this was also performed in a kind of simulation mode, while the remainder of the formal crew of six 
prepared dinner and wrote reports (and others returned to Hanksville for the night). 

Our objectives this year were to demonstrate longer duration (> 30 minutes) and more distant (> 100 meters) 
EVAs with the Mobile Agents system, including: 

o the EVA Robotic Assistant 18,19 (ERA, http://www.jsc.nasa.gov/er/era) integrated with astronaut operations 
o more sophisticated voice commanding for science data collection (images, voice annotations, and samples) 
o automatic association of science data with named (and automatically mapped) work sites 
o appropriate alerting for exceeding health, schedule, and location thresholds, including verbal warnings to 
the astronauts, as well as on the loudspeaker in the hab, and via email to the RST 
o collaboration with a well-organized remote science team 
o planning and meeting replay tools for crew-RST communications 
o EVA data stored automatically in a database shared by the crew and RST. 

Our progress over the two weeks was recorded in daily reports ( http:/Avww.marssocietv.ors/MDRS/fs03) . We 
were host to several NASA managers, two video crews, and an international reporter team. During the second week 
the field experiments were presented on NASA TV’s Video File. 

The schedule was deliberately aggressive, with a week for each of two major sites. However, we had planned to 
take the hab into “closed simulation mode” after EVAs to document sample curating and mission planning and only 
did that once. The second week’s EVAs ended at such odd hours (sometimes after dark) and we were so tired, 
continuing our work in a formal way was impractical. Closed system simulation of planning, EVA, and review will 
have to wait for a more routinely usable system, certainly one not requiring a dozen support people and a mini-camp 
in the field. Nevertheless, Sierhuis always carried through the sequence of interactions with the RST (organized 
remotely by Rupert), and this was itself almost a second expedition on the side, with many participants, tools, and 
reports. We learned that having EVAs on sequential days was difficult to cooperatively plan and review, given the 
uncertainty of testing and multiple time zones. Detailed reports about the scenarios appear in the main sections of 
this paper. See Table 2 in the Results and Discussion section for an example “run” of the system. 

B. Annual Cycle of Analog Research and Development 

From a scientific perspective, an analog is any site (e.g., deep ocean hydrothermal vents) or object (such as a 
model in a wind tunnel) that can be studied first hand to learn about another place usually remote or impossible to 
study directly (e.g.. Mars) or an artifact that is being designed (e.g., a space vehicle). In space exploration, analog 
sites were commonly used in Apollo for training astronauts (see Ref. 11 for a survey and the references in Ref. 9 and 
10). Today, scientists, engineers, and flight controllers work together in analog sites for experimenting with tools 
and operations concepts. For example, analog sites were used to prepare for the Mars Exploration Rover mission 
during 2003. 

The use of analog missions is thus a research and development methodology that integrates operations and 
technology research. “Research” is used more broadly here than is familiar to some managers — just as the 
collaboration of flight operations practitioners and academic researchers may be unexpected. Research here refers to 
any scientifically studied phenomena in authentic work settings, to reveal the interactions of practice (e.g., how 
geologists prefer to explore), technology (e.g., a robotic system), and operations (e.g., protocols for communication 
between the crew and RST). Research of this kind, including observational and analytic studies, is an essential 
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aspect of work systems design, which specializes in practically relating roles, procedures, facilities, schedules, and 
tools. 12 

Recently, computer scientists and flight operations researchers have chosen to follow geologists and biologists to 
Mars analog settings, where exploration can be studied and technology used experimentally. 20 ' 22 After several years, 
we can identify an analog learning cycle, which iterates through at least five phases: 

1 . Analyze Data (including previous field experience, historical analogs, and related work). 

2. Identify Operations Concepts (including new scenario and technology opportunities). 

3. Requirements and Design Workshop (>= 6 months prior to field work, including all project 
participants). 

4. Write Buildable Specifications (constrained by time, funding, personnel; including effort and time 
worksheets) 

5. Operational Readiness Test (at least one month prior to field test, bringing key personnel and 
equipment together for connectivity and communications tests in a conventional laboratory/research 
center setting). 

6. Field Experiment in Authentic Work Setting (two weeks or more, with convenient living and work 
facilities, including observation and recording of simulated operations). 

Following this cycle since early 2002, the Mobile Agents project has progressed from connectivity trials to nearly 
practical application of the tool during actual geologic exploration. Figure 1 shows how the use of the tool 
progresses in phases from mere testing (which does not require authentic work settings) to exploration in settings 
that are relatively well-known to science, but new to the participants, to exploration**** that involves authentic 
science (new, publishable discoveries). For most research and development purposes, analog experiments need only 
involve authentic exploration; however, some locations such as the Arctic and Antarctic offer the opportunity for 
new geology and biology research work as well. tttf Note that Fig. 1 can be adapted for engineering work (e.g., 
systems monitoring). Engineering scenarios and configurations are being developed for Mobile Agents field 
experiments in 2005. 



Figure 1. Relation of science, exploration, and analog experiments. 

Research in analog settings investigating a particular work system design shifts from 
1) controlled tests (in which the only authentic experience is using the tools in 
largely predefined settings) to 2) scripted exercises (involving authentic exploration) 
to 3) planned exploration goals and regions (involving authentic scientific 
exploration). “ Science ” here refers to geology, biology, astronomy, physics, etc. The 
scientific study of people and work systems for the design of space missions proceeds 
in parallel on another dimension. 


“Exploration” in the context of space exploration involves getting to know a place, with typical phases of 
surveying, mapping, route finding, and region/boundary demarcation. 

Note that the word “science” is used here quite narrowly, which is common in the space exploration community, 
to connote the scientific study of life and the universe. In practice, the development of exploration tools involves 
new scientific research in many other disciplines, including anthropology, psychology, sociology, and of course 
computer science. More broadly, the design of flight operations systems involves a kind of scientific work that 
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Furthermore, in the Mobile Agents project, we are investigating the use of simulation-based tools for designing 
field configurations and scenarios. 2 ^ 29 Such simulations could become part of the analog learning cycle, such that 
simulations are fine-tuned from experimental results, and potentially used for evaluating designs when analog 
experiments are not practical. A simulation using the Brahms tool can be transformed relatively directly into a 
runtime system, such that some computer agents are replaced by actual people and robots, and others become part of 
the software used by the astronauts, mission support, engineering, and science teams. 

An important consideration in Mobile Agents system development is the relation of the tools and scenarios to 
NASA’s exploration missions. As mentioned, it will take decades before we accomplish the vision of automating the 
role of Capcom. However, near-term applications can be tied to the requirements for the Crew Exploration Vehicle, 
emergency EVAs in space, and construction/maintenance in the next lunar expeditions. In this process, we relate the 
overall work to the MA design in three broad levels of analysis: 

1. Identification of Exploration Activities and Tasks (e.g., communications between remote science team 
and ISS crew). 

2. Determination of Fixed Constraints (e.g., Terrain, Timing, Crew Size, External Support) 

3. Specification of Design Parameters, for example: 

• Data and workflow systems 

• Tools: Drills, construction equipment, instruments 

• Suits, gloves, life support 

• Transportation 

• Operations protocols: Crew (vehicle/habitat and EVA), Support (engineers and scientists) 

Often, in developing tools for future missions, we proceed bottom-up, so we can establish a basic technology 
infrastructure. However, we expect that soon analog research and development will become more “middle-out” as 
we relate problems posed by actual mission requirements to the opportunities afforded by existing analog sites, 
facilities, and technological capabilities (see the Conclusions section). 

C. Scenarios and Schedule Overview 

The 2004 field experiments were organized for two distinct settings, one week each. Each setting required 
completely different deployment of the wireless network backbone, with 1-2 days required for reconfiguration. Each 
setting involved multiple days of work, with different scenarios involving very different interactions between the 
two geology “astronauts”*** 1 and the EVA robotic assistant. The scenarios are reviewed briefly here; more details 
appear in subsequent sections. 

The first week involved a pedestrian EVA to “Pooh’s Corner,” a site of interest to the geologists, identified by a 
brief visit in civilian clothes (called a “baseline study”) in 2003 (see Fig. 2). On the first day, the ERA performed an 
“autonomous” seven hour reconnaissance, following a route with predetermined waypoints and transmitting video, 
photographs, and panoramas along the way. An EVA plan was developed using data returned by the ERA and 
available aerial maps. The plan was represented using computer tools, shared with and annotated by the remote 
science team (RST), and then uploaded into the Mobile Agents system as a sequence of located, timed activities. On 
the next day, the astronauts explored on foot and sampled locations according to the activities represented in the 
plan. The plan provides expectations to the Mobile Agents system, such that it monitor the schedule and the 
astronauts’ location, providing appropriate alerting to the crew on the EVA and in the hab. 

The second week experiments occurred at “Lith Canyon,” a site 5 km away, also involving multiple scenarios. 
On one day, the ERA was left at the head of the canyon, moving between predetermined waypoints and transmitting 
photographs and video that enabled the habitat crew to monitor the astronauts in the canyon. On a much longer five 
hour EVA, the ERA was lowered by block and tackle into the canyon, so it could accompany the astronauts — 
serving as a network relay, carrying samples and tools on a cart, plus taking photographs and panoramas on voice 
command, as well as video-tracking an astronaut. The ERA successfully followed the astronauts approximately 150 
m, at a rate of 0.15 m/s with laser-based obstacle avoidance (0.5 m/s without obstacle avoidance on relatively flat 
terrain). 

The voice commands introduced for 2004 included: 

• Boudreaux 5555 , follow/watch/take a picture of me 

• Boudreaux, take a panorama 


cannot be reduced to any of these disciplines (e.g., see the Results and Discussion section for a summary of 
requirements discovered empirically during Mobile Agents 2004 experiments). 

**** We will refer to the geology graduate students as “astronauts” so the roles being played are clear. 

§§§§ jhe ERA team has developed multiple robots. Experiments in 2004 used the ERA named “Boudreaux.” 
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• Boudreaux, execute your plan/come home 

• Label the last image/collection as X 

• Start activity walk/work/survey/sample {at location Y/here} {for time T} 

• Change the duration of the current activity to X seconds/minutes/hours 

• Associate voice note/image/sample bag {called X} with {the last} sample bag/image/voice note {called 

Y} 

• Create sample bag LL/DD/DD tfttt 



Figure 2. MDRS Topographic Setting and Scenario Relay Configuration. R1 designates the 
antenna on Pooh ’s Corner hill, providing a wireless LAN relay in first week’s scenarios. R2 and R3 
designate antenna placement on ridges ("WLAN Backbone") for Lith Canyon scenarios in the second 
week, approximately 5 km from MDRS hab. An ATV served as an access point on the opposite side of 
the canyon, with the ERA following the astronauts providing the final relay on the canyon floor. A 
TES trailer-based satellite system, connected by Ethernet cable to MDRS, provided internet access to 
NASA Ames, where the science data were mirrored for access by the remote science team. 

• Call my location work site DD 

II. Participants in Mobile Agents 2004 Field Experiments 

Over fifty people participated in this project during the two week 2004 field experiments. Twenty-two people 
worked on-site at the Mars Desert Research Station, including the crew who lived in the hab and provided regular 
outreach reports (Clancey, Sierhuis, Alena, Dowding, Garry, Semple). These people came from three NASA centers 
and six universities: 


***** Names in the 2004 configuration must be one of a set of pre-determined vegetables (e.g., “asparagus”) or by 
saying “work site” followed by a number. 

L is a letter specified by a predetermined vocabulary, e.g., “Whiskey-Lima” = WL, which designates “west 
Lith”; D is a digit (0 - 9). 
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• NASA Ames Research Center (Moffett Field, CA) 

o Brahms Modeling and Simulation System ( www.agentisolutions.com') : Clancey and Sierhuis, Co- 
PIs; Ron van Hoof, Michael Scott, Yilmaz Cengeloglu, Charis Kaskiris (UCB), Jeff Bradshaw 
(Institute of Human and Machine Cognition, University of West Florida, Pensacola, FL) 
o Mobile Exploration System (MEX, wireless computing and communications system): Rick Alena, 
PI; Charles Lee, John Ossenfort, Ed Walker 

o R1ACS Language Interfaces and Speech Technology group (RIALIST; 

www.riacs.edu/speech/sDeech.htmn : John Dowding 
o NASA Research & Education Network (NREN ; www.nren.nasa.govA : Ray Gilstrap, Thom Stone 

• NASA Glenn Research Center (Cleveland, OH) 

o Michael Cauley, David Pleva, Mark Seibert 

• NASA Johnson Space Center (Houston, TX) 

o EVA Robotic Assistant (ERA, aka “Boudreaux” and “the robot”): Jeff Graham, PI; Kimberly 
Tyree, Nathan Howard, Robert Hirsh 

• SUNY Buffalo (State University New York at Buffalo) 

o Abigail Semple, Brent Garry (Geology Department) 

A group of at least 16 more people were part of the Remote Science Team, led by Rupert and Sierhuis: 

• Remote Science Team — Geologists and Biologists 

o Shannon Rupert (MiraCosta College, Oceanside, CA) 
o Stacy Sklar (Northern Arizona University) 

o Melisa Farley, Kyle Frederick, Brett Burkett, Shannon Kobs, Nadine Calis, Amy Webb (SUNY 
Buffalo) 

• Remote Science Team — Collaborative Tools Support 

o Compendium-. Simon Buckingham Shum, Michelle Bachler, A1 Selvin (Knowledge Media Institute, 
Open University, Milton Keynes, UK) 
o BuddySpace : Marc Eisenstadt and Jiri Komzak (Open University) 

o Meeting Replay tool'. Danius Michaelides, Kevin Page, Dave De Roure and Nigel Shadbolt 
(Intelligence, Agents, Multimedia Group; Univ. of Southampton, UK) 
o ScienceOrganizer: Dan Berrios, Ian Sturken, David Hall, Rich Keller (Code IC, NASA-Ames 
Research Center) 

Other support behind the scenes was provided by NASA Ames Public Affairs, the RIACS administration at 
Ames, and the Northern California Mars Society volunteers who provided logistic “mission support” (plus Tony 
Muscatello in Colorado, representing the Mars Society). Frank Schubert and Don Foutz (owner-manager of the 
Hanksville Whispering Sands Motel) maintained and resupplied the utilities (power, water, septic, ATVs). 

III. Mobile Agent 2004 Configuration 

Here we describe the configuration of software and devices, called the Mobile Agents Architecture (MAA), used 
for the 2004 field experiments. The MAA changes from one year to the next, with different configurations used for 
scenarios within a given field season. For example, during the Pooh’s Comer reconnaissance in 2004, the astronauts 
were inside the hab and not giving voice commands to the robot; instead they monitored its location and the returned 
data on computer screens. Each MAA configuration consists of revised capabilities for the existing software agents, 
new software agents, new software systems (e.g., crew planning tools), new voice commanding vocabulary and 
processing, and new devices (e.g., a panorama camera). 

A. Background: Software agents and Multiagent Systems 

The term software agent is used in different ways in computer science. Software agents, also called “intelligent 
agents,” were invented in the 1970s by artificial intelligence researchers as programs that could assist computer 
users, such as for finding resources on the internet. 30 A software agent is essentially a computer program with goals, 
models, inferential capability, and a memory of “beliefs” (propositions about the world), designed so people can 
interact with it as if it were an independent entity. Starting in the 1980s, this approach became particularly useful 
when software agents were able to communicate over the internet, serving as “personal agents” 31 that manage the 
interests of different people. 

In the 1990s, the software agent representational approach was strengthened by the advent of multi-agent 
systems, essentially systems of coordinated programs within some network and often geographic setting. 32 With the 
invention of internet browsers and their nearly universal use for commerce and research, plus the availability of 
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much faster computers with more memory, the multi-agent approach became adopted for a variety of practical 
applications. In the Mobile Agents project, we go a step further by implementing the agents on portable computers 
on a wireless network. In considering the benefit of software agents in the design and implementation of the Mobile 
Architecture, especially as a means of integrating a diversity of systems with people in a work system, we believe 
that the software agents approach is not simply a way of programming, but a new paradigm for software 
development. 


B. MAA Technical Overview 

The Mobile Agents Architecture consists of four software foundations: the Java programming language, the 
Brahms multiagent modeling and programming language, 23 ' 24 the KAoS agent-communication framework, and 
CORBA. Together these software tools allow us to implement the MAA distributed over a wide variety of 
wirelessly networked computers, from the astronaut spacesuit computers, to the robot computers and the computers 
in the MDRS habitat, to the computers for the Remote Science Team (RST) back on Earth (Fig. 3). 

In all that follows, the reader should remember that no particular MA configuration represents a recommendation 
of “what should be,” but rather is always a compromise for what funding allows, what time and personnel available 
permit to be built and tested, and the technology available and convenient to deploy. What is important, for example, 
is not the use of USB for connecting a camera to a computer or the use of GPS, but the concept and operation of the 
multi-agent system as a means of managing exploration, scientific, and engineering work in logistically challenging 
environments. The essential elements of the MAA are: multi-layer communication network 34 (i.e., LAN, agent 
registry, distributed agents); distributed computers communicating with each other and attached peripherals; use of 
APIs to integrate peripherals with the agent system; personal agents managing workflow (including agent 
monitoring, advising, and data management); crew commanding and feedback protocol^ 5 automated association of 
time, location, and activity data with science data; 36 activity-based planning (vs. task-based approach 26 ); tools to 
facilitate remote collaboration between crew, support, and RST teams. 
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Figure 3. Mobile Agents 2004 Network and Components Configuration. Notice that the ERA serves as relay 
to the astronauts, which enables extending EV As around corners and into craters or ravines. 


Figure 3 shows the hardware, network and software elements in MAA 2004. Starting at the right side of the 
graphic we see the two EVA astronauts with their backpack computers. Each backpack computer runs the Brahms 
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software agents that integrate and communicate with ail the other software pieces in the architecture. The agents 
coordinate the dataflow between the systems as well as the work process of the astronauts. The dialogue system 
software allows the astronauts to speak to their personal software agents using speech in a predetermined vocabulary 
with flexible phrasing. The GPS system gathers location data from differential GPS (dGPS) units with 2 cm 
accuracy. The biovest system gathers astronaut health information from health-monitoring sensors. Digital cameras 
can be connected to the backpack computers so software agents can download digital image files taken by the 
astronaut and associate them with other gathered data (such as the astronaut's GPS data), as well as store and 
forward these images to the habitat and Earth. 

Next, one of the ERA computers runs the Brahms software agents' supporting the ERA in executing plans and 
allowing the robot to communicate its information to the Brahms software agents on the astronaut’s spacesuit and in 
the habitat. The ERA software agents allow us to easily integrate the ERA’S capabilities with the rest of the software 
in the system. The ERA can be used in two different modes, an autonomous mode and an interactive team mode. In 
the autonomous mode an ERA software agent executes a previously uploaded EVA plan. This allows the robot to go 
on EVAs by itself and take panorama and other images that are immediately communicated back to the habitat. In 
the interactive team mode the ERA supports the astronauts on their EVA and gets its commands either via a speech 
command from the astronauts or a crew member in the habitat (called the habitat communicator, HabCom), or from 
a graphical user interface (GUI) operated by HabCom. The ERA can take digital panorama images, find its way to 
named GPS locations, carry the astronaut’s tools and science samples, print a curation label for the rock and soil 
samples gathered by the astronauts and autonomously follow an astronaut. The ERA robot can also function as a 
network relay node for the astronauts, allowing the astronauts to wander to interesting areas providing network 
connectivity back to the habitat. 

An all-terrain vehicle (ATV) serves as a wireless network “hub” between the astronauts, the ERA robot and the 
habitat (Fig. 3). Using this approach we can reconfigure the communication path for the dataflow between 
astronauts, robot and habitat. The ATV computer serves as the MAA agent directory service, allowing the agents to 
find each other on this distributed network. The KAoS software that is part of the Brahms agent software 
environment in effect creates a distributed agent framework allowing the MAA software agents to be distributed and 
communicate over a 5 km wireless WAN. 

Inside the MDRS habitat a set of software agents serve as the central coordination, communication and storage 
facility integrated with a number of other software systems previously developed to store science data, and allow 
distributed teams to collaborate asynchronously (this is important because the communication time-delay between 
Earth and Mars is too long to permit synchronous collaboration). This integrated habitat software environment 
allows the crew to collaboratively, but asynchronously, create EVA plans with the RST and analyze science data in 
the habitat after an EVA. Software agents running inside the habitat automatically monitor and store science and 
EVA plan data that is communicated by the software agents of the astronauts and ERA. After storing this data, a 
software agent automatically emails EVA data to the RST back on Earth via a satellite connection to the internet. At 
the same time the data stored inside the habitat's ScienceOrganizer system 36 is automatically copied to a 
ScienceOrganizer database back on Earth. This allows the RST to access the data in near real-time (with an artificial 
five minute delay). EVA plans are created and communicated by the crew to the RST using the Compendium 
software tool (http://www.compendiuminstitute.org) that is specifically developed for discussions and modeling 
problems in the form of hypermedia concept maps. This Compendium tool is used to allow the crew and the RST to 
collaborate. A Compendium software agent in the habitat can read the EVA plan from the Compendium database 
and distribute the plan amongst all the software agents in the MAA. This agent also stores all science data that 
comes into the habitat in the Compendium database, so that the crew and the RST can discuss the data within the 
Compendium tool. 

Figure 4 gives a graphical view of the software agents within the MAA. The solid icon circles in the three black 
ovals show the software agents running inside the Brahms virtual machine on the astronaut’s backpack, the ERA 
robot and on the HabCom computer inside the hab. 

Figure 5 presents an example of the information flow using proxy agents (a form of interagent communication 
used in the 2002-2004 field experiments). Astronaut one says that he is starting the activity of sampling an area. This 
is processed by Astronaut one’s dialog agent and passed as a communication to his personal agent (PA). From here, 
the information needs to be communicated to the HabCom PA, which occurs via the HabCom PA proxy that co- 
exists in the Brahms system running on Astronaut one’s backpack. The proxy uses information provided by a central 
registry (in this Mobile Agents configuration) to find the HabCom PA running on a platform in the hab. From the 
HabCom PA, the information is communicated by email to the RST and by a loudspeaker to the HabCom (a crew 
member in the hab) via the HabCom dialog agent (which converts the text to speech). HabCom PA then provides 
feedback to Astronaut one and two that the activity has begun, first using proxies to find the corresponding PAs, 
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from which the information is communicated to the corresponding dialog agents, and then spoken to the astronauts 
on their headphones. All of the transmission of requests and scientific data follows similar paths between the crew 
members and their PAs. 

Note that biosensor and GPS data is handled more directly for efficiency by pulling data as needed from external 
sources, rather than using the agent communication network (which is to say that agent communications do not 
replace telemetry networks). Furthermore, there is nothing fixed about the design of proxies and PAs shown here; 
this is just one of many possible ways to partition computational procedures and information flow. See the Results 
and Discussion section for related comments about improved designs. 
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Figure 4. Brahms software agents in the Mobile Agents 2004 Configuration. 
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IV. Crew Collaboration with Remote Science Team 

The small Mars crew will undoubtedly collaborate with groups of scientists back on Earth. How this 
collaboration will happen is a matter of conjecture and experimentation. The Mobile Agents framework provides a 
means for implementing a computer-supported Mars- and Earth-based science work system, which we first 
employed in 2004. This system includes both the human work practices and computer tools with dataflow 
management systems. Here human-centered design meets work process design. In the empirical design approach we 
are using in the Mobile Agents project we are guided by the capabilities of the people and their objectives. People 
are at the center of the total system, and people are supported in their work by computer tools. This can be 
contrasted with a top-down functional approach, which attempts to “optimize” the work to be done, automating as 
much as possible, developing corresponding tools, and then training people to deal with the resulting work processes 
and interfaces. Instead, we develop automation that supports the human work practices, allowing people to do what 
they do well, and to work in ways they are most familiar. 

We start simply, asking basic questions such as how the Mars crew can communicate their daily EVA plans and 
captured science data during and after an EVA back to their colleagues on Earth. This leads us to question what the 
role of an Earth-based science team should be. Can the RST participate in the planning of daily EVAs on Mars? Will 
the RST be able to get the science data in time to make useful suggestions to the crew? Will the RST be able to 
follow the field crew’s investigations? Will the crew be able to absorb the RST suggestions in a timely manner to 
develop a daily EVA plan? How will the RST EVA plan compare with the crew plan? 

For the initial experiment in 2004, we defined a relatively simple science work process, using three pre-existing 
domain-general software tools. We integrated two of the three tools with the MAA, enabling the automatic flow of 
data between Mars and Earth. With this integration we showed that it is easy to add new software tools to the MAA. 
Previously developed software tools and systems can interact with software systems already in the architecture 
without having to develop dedicated software interfaces between them. This important feature of the MAA allows us 
to freely include separate tools to support specific functions in the work process. 

We briefly describe the make-up of the RST and the crew/RST workflow process. We then describe how the 
three tools support the crew and the RST in doing their work. 

A. The Remote Science Team 

The RST for the 2004 field experiments was highly distributed, consisting of scientists from all over the world. 
Figure 6 shows the RST organization. The team had four parts, one located in San Diego, New York, Arizona, and 
England. Rupert coordinated these parts, plus was point of contact between the Crew Uplink Lead (Sierhuis) and the 
RST. The SUNY Buffalo team consisted of faculty and geology graduate students from the Volcanoes Studies 
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Figure 6. The Remote Science Team. Mars Crew inset corresponds to Fig. 4. 


Group. Sklar, a student at Northern Arizona University, led a team made up of geologists from the MDRS RST, 
volunteer scientists who supported all of the 2004 MDRS rotations and have participated as field crew. Buckingham 
Shum and Selvin of The Open University in the UK led the meeting facilitators, who managed communication 
between the crew and the RST through Compendium and Meeting Replay. 

The RST teams collaborated to analyze the EVA science data and provide feedback on the crew’s next EVA 
plan. We defined a simple work process in which the crew planned and executed their EVAs in a daily EVA-cycle 
using several tools. Because the RST members were so distributed, their interaction also benefited from 
collaboration tools. The RST Facilitator facilitated web-based teleconferences between the RST teams. The 
workflow process is described next. 

B. The Crew/RST Workflow Process 

The workflow process is divided into two parallel processes that are dependent on each other. Figure 7 shows a 
high-level workflow process model. 

The idealized flow shown in Fig. 7 is as follows: After the EVA, the crew analyzes the collected science data 
(rock and soil samples) back in the hab. After this analysis, the crew discusses the results of the day’s EVA and plan 
the EVA for the next day. This meeting is facilitated with the Compendium meeting capture software and is video 
taped. The result is communicated to Earth. On Earth, the crew’s meeting video and Compendium database is used 
in the Meeting Replay tool. This allows the RST to view the EVA planning discussion of the crew and at the same 
time view the crew’s plan in the Compendium tool. After the individual RST members have viewed the video, they 
have a remotely facilitated meeting on the web, called the Science Operations Work Group meeting. The name is 
taken from a similar meeting held during each sol on the Mars Exploration Rover (MER) mission at JPL 
(http://origin.mars5.jpl.nasa.gov/home/). During the SOWG meeting on the MER mission, science subgroup leaders 
review and assemble the plan for the next day, relating timing and resources. Similarly, the RST team meets together 
in a teleconference to review the plan made earlier by the crew on Mars. This meeting is also captured in the 
Compendium meeting capture tool. The result is emailed back to Mars before the crew wakes up the next morning. 
The next morning at 8:30 (local Mars time) the crew reviews the RST plan in Compendium, and decides on the final 
plan for the day’s EVA. 

More details about the collaboration tools are provided in the following section. 
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Figure 7. The Mobile Agents Project Crew/RST Workflow. 


V. Crew/RST Collaboration Support Tools 

Mobile Agents 2004 introduced a new strand of research in MDRS analog research, concerning how to support 
the Remote Science Team (RST), which in actual missions is likely to be spread across the Earth in multiple time 
zones. Just as the MER 2004 mission showed after a few months, it is impractical to physically co-locate all relevant 
experts who might be needed in a multi-year Mars mission. Looking ahead a decade or two, this research prototypes 
and evaluates tools to support scientific teamwork under such circumstances, advancing the state of the art in 
information networks. 

To this end, a collaboration was formed between NASA-Ames’ work systems design research group and the 
UK’s CoAKTinG Project. 37 CoAKTinG (“Collaborative Advanced Knowledge Technologies in the Grid”) is part of 
the Advanced Knowledge Technologies consortium and is developing new kinds of collaboration technology to 
support what they call e-Science. MA 2004 provided a testbed for some of the CoAKTinG tools, to see how well 
they could connect the RST with each other, with a crew living and working out of conversational contact. 

A. Preparation for Field Experiments 

To prepare, the RST held weekly planning meetings via teleconference for over five months, exploring possible 
science questions that could be addressed by the field crew during the field test and putting the new collaboration 
support tools through their paces. Thus we began to understand the role that different collaborative technologies 
could play in the overall work ecology of the RST and astronauts. 

In particular, during the preparation the RST used the ScienceOrganizer system to archive and to link 
meaningfully their project materials and experimented with Compendium to design their own maps to support a 
geological methodology. A trial version of the Meeting Replay tool from the Operational Readiness Test in March 
2004 also generated much interest, and was fine-tuned as a result. The BuddySpace messaging and awareness tool 
was released once the MDRS work had started. These RST Communication System tools are described in 
subsequent sections. 

B. ScienceOrganizer: Long-term, formal memory for mission data and knowledge 

ScienceOrganizer 36 was developed at NASA Ames, and provides a web-based semantic database that can be 
populated and searched by scientists or engineers. It has been previously used to support the Columbia Shuttle 
accident investigation, 38 to study patterns of malaria in Africa, and to design and execute experiments on microbes, 
among other applications. ScienceOrganizer provides a rich interface for semantically linking resources (i.e., a 
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relational database with well-defined descriptive relations, such as “plan activity”), and is most suited for long-term 
knowledge management, especially for distributed teams collaborating asynchronously. Figures 8 and 9 illustrate 
how a voice note is stored in ScienceOrganizer and the notification is received by a member of the RST. 

From: capcom@agentisolutions.com 
Subject: MDRS New VoiceAnnotation: voice_note_9 
Date: May 7, 2004 3:47:36 PM PDT 
To: mdrs rst@agentisolutions.com 

New VoiceAnnotation: voice_note_9 

EVA Plan: LithCanyon SegmentThreeEVA Plan 

Activity: WorkAtWayPointl6 

Creator: AstroTwo 

TimeStamp: 05/07/2004 23:47:12 

File Name: voice_note_2004-4-7_23-46-28.wav 

Latitude: 3827.3077 NORTH 
Longitude: 11047.4074 WEST 

Northing: 4256335.2693082085 Easting: 518312.1563357575 Zone: 12S 

ScienceOrganizer Link : https://marst.arc.nasa.gov/org/166015 

Figure 8. Email sent to RST indicating a new voice note. Besides the time and precise GPS location, the email 
indicates that astronaut 2 created the voice note while working at waypoint 16 during the Segment 3 EVA in Lith 
Canyon. This information is associated with the voice note in ScienceOrganizer, at the specified URL. 
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Figure 9. A ScienceOrganizer database entry showing a voice annotation created Lith Canyon EVA. 

On the left are hyperlinks to related resources, such as the location the voice note was taken, the astronaut, and the 
EVA itself. This is the page referenced in the email sent to the RST shown in Figure 8. 
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Figure 10. An EVA plan constructed by the Crew in Compendium. 

This plan was read by the Brahms software agents that coordinated the EVA work flow. 


2. Compendium was used to view all the data gathered on the EVA when the Crew returned to the hab. Ail 
the data generated during an EVA (videos, photos, voice annotations...) streamed back over the wireless network 
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and were stored in the hab in ScienceOrganizer and Compendium. Compendium displays data in concept maps (Fig. 
11), assisting analysts in seeing and navigating certain kinds of structure, and linking in their own ideas and 
documents. Compendium maps were sometimes shared by storing then in the ScienceOrganizer database, and data 
referenced by maps were archived in ScienceOrganizer. 
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Figure 11. Following the EVA, a map is created and populated in Compendium rendering the data 
gathered. Shown is an example Science Data map, highlighting the data gathered by Astronaut One. 


3. Compendium was used by the crew and RST to discuss the implications of the data. Compendium 
displays information in visual concept maps to support sense-making activities such as planning, discussions, and 
the gradual structuring of ideas. As the discussion unfolds, the contributions are simultaneously mapped on the 
screen (projected in the hab in a crew meeting, or shared over the internet in an RST teleconference). The RST could 
therefore see what options the crew may have discussed but rejected, and why, in order to provide scientific 
feedback to the crew (Fig. 12). 
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Figure 12. Compendium map summarizing the RST’s feedback to the crew. Each node is hyperlinked to the 
map (i.e. the context) in which it was created. 


The insight that the RST gains into the crew’s decision-making is further enriched by an indexed, digital video 
record of the crew’s meeting which integrates Compendium with other resources, as described next. 


D. Meeting Replay: Enabling the RST to review a crew meeting 

Given the communication delay between Mars and Earth, the usual electronic ways of working together at a 
distance, such as conversation and the sharing of computer screens, are impractical. As part of the CoAKTinG 
project, the University of Southampton have developed a meeting replay tool, which combines meeting materials 
within an interface structured to enable quick and easy indexing navigation of the meeting record. 

During the mission we recorded the crew’s daily EVA planning meetings and delivering a replay of the meeting 
to the RST within a few hours. By experimenting with these techniques we hoped to see if the RST could gain a 
better understanding not only what a crew is deciding, but why, and how, in order to provide the best kind of 
feedback. 

Figure 13 shows the web-based Meeting Replay tool. The upper region shows the video of the meeting and the 
Compendium map as the discussion progresses. The lower region contains summary information about the 
meeting — who was there, who was speaking, the agenda, and an overview of the current topic (derived from the 
Compendium map). Some of this information is presented as a timeline, providing a visual index for an RST 
member to navigate the video, jumping to relevant or interesting parts of the discussion by clicking on the timeline 
or moving the slider. 


17 



23 [Map]: Paah3 Comer DayOneEva Crew Planning Meeting 04/27/04 



Where should Boudreaux rake 
Panoramas? 


Where is Pood’s Corner? Looking at the Aerial photo 1 m resolution 


Poohs ^Corner JWaypomC _14 


<D 


K takes about an Ikjut for Boudreaux to 
gettoWP 10 



How long do we have? 

I 

We can do 10 at the entrance. 


We have approx. 3-4 hours of battery 


CD 


Title: MOPS' Day One EVA Debrief Meeting 4/27/2004 
Date: Wed Apr 28 01:51 32 2004 
Participants: Maarten . Brent. Abigail . Richard. John . 


'Current Speaker: Maarten 

Nodes: 1 : 1?Go t 0 the panoramas and go over the naming scheme 



Kkkwd 


Video 

GioupSync QOniim 

Mods MaoterQ ® Stave 
Receiving Yea C 0 .Vo 


Figure 13. Web-based meeting replay tool screenshot. When reviewing the meeting replay, Compendium has 
been extended so that it can be used as a ‘visual contents page ' into the video. For instance, if the RST wants to see 
discussion prior to the recording of a particular decision, one can now click on this node in Compendium and the 
replay jumps to the point in the meeting where that node was recorded, and starts to play the video. 

Before the RST meets collectively, each participant runs their own meeting replay. Should the need arise to 
review a specific point in the meeting as a group, one member can take control of all the meeting replays using the 
GroupSync feature. As that member navigates around the meeting everybody else’s client follows their lead, 
remaining in synchronization as long as required. 


E. BuddySpace: Instant messaging meets presence visualization 

Teams distributed in time and space are less aware of their colleagues’ activities and availability, giving a 
reduced sense of presence compared to being co-located. One challenge is how to characterize presence, as 
something that can be managed and visualized easily, while remaining consistent with the user’s expectations and 
work habits, including existing patterns for using Instant Messaging (IM) and other communication tools. 

A prototype presence management tool called BuddySpace has been developed at the Open University, which 
overlays presence information onto visualizations, both geographical (e.g. a map of a building, region or planet), and 
conceptual (e.g. a workflow chart, project plan, or experiment). (BuddySpace is based on Jabber/XMPP, and is 
interoperable with MSN Messenger, Yahoo, ICQ, and AIM.) In the RST BuddySpace map (Fig. 14), the availability 
of members is shown mapped to their geographical location. 
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Figure 14. Prototype RST BuddySpace map, in which the online availability of colleagues is superimposed on 
locations in the US and UK. Clicking on one of the presence dots opens a conventional instant messaging session. 


The preparation and simulated EVA experience suggested that ScienceOrganizer, Compendium, Meeting 
Replay, and BuddySpace used together is a powerful suite of tools for supporting RST-Crew collaboration. 
Reporting at a mission web site provided important means of communication between the crew, RST, mission 
support, and the public. These reports are described in the next section. 


VI. Reporting During the Simulated Mission 

The Mars Society maintains an archived set of crew reports throughout the multiple year campaigns at the 
Flashline Mars Arctic Research Station (FMARS) and MDRS. Each crew page includes biographies, mission 
information, and daily reports with photographs. A good part of this article was prepared from reports written and 
posted during the two weeks field experiments (see http://www.marssociety.org/MDRS/fs03). 

Each member of the official MDRS29 crew wrote reports according to his/her role: Commander “check in” 
reports provided daily synopses about the status of system deployment and testing, as well as a window on living 
and working at MDRS during this period. More complete EVA reports were written by the geology-astronauts. An 
EVA Communications Systems Report described the interaction of the crew with the RST and other aspects of the 
field work. Other crew reports included: Engineering (e.g., generator status), Greenhab (a gray-water recycling 
system), plus Health and Safety. To complement these reports, the RST Communication System Report was written 
by the RST Facilitator (Shum) in the UK, describing the experience and status of the collaboration tools. The RST 
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report written by the RST Lead (Rupert) detailed the experience of the RST team in understanding and commenting 
on EVA plans and data. 

Some of these reports are excerpted and adapted in subsequent sections, which describe the primary scenarios of 
MA 2004 and what occurred. 

VII. Pedestrian EVA Scenario (Pooh’s Corner) 

The first scenario involved several sequential activities: Reconnaissance by the ERA, crew-RST planning, and a 
pedestrian (walk out of the hab) EVA. These activities were planned for two contiguous days, though weather 
required a break in between. 

A. Day One of the Pooh's Corner EVA: ERA Reconnaissance and Crew Planning 

The first day plan was to send Boudreaux (the ERA used in 2004) over the previously planned path to Pooh’s 
Corner to take some panoramas. After we looked at the available aerial image of the area around the hab that 
includes Pooh's Comer, we decided that we would create a plan that would automatically drive Boudreaux to Pooh's 
Corner and take four panorama images of the area. Figure 15 shows the discussion mapped in the Compendium tool 



Figure 15. Morning briefing notes and plan in Compendium, for first day Pooh's Corner ERA 
reconnaissance. 


on April 26, including a sketch of the plan for the ERA’S operation the next day. 

The actual EVA plan was also created in our Compendium meeting capture tool (Fig. 16) from waypoints that 
had been previously determined by walking through the site and logging GPS locations (work planned for 2005 will 
seek to eliminate this step, which of course contradicts the overall notion of “autonomous” robotic reconnaissance). 
After the waypoint plan was created in Compendium, it was automatically translated into a MA plan that could be 
interpreted by the ERA’s personal agent. 
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Figure 16. Plan created for ERA’S reconnaissance at Pooh’s Corner. Each waypoint icon designates an object 
containing additional information about location and actions. This representation can be automatically interpreted 


by a program that generates a plan represented in Brahms that is used by mobile agents during the EVA. 


The Mobile Agents system was 
started, the plan was read in by the 
HabCom agent and distributed to the 
ERA Personal Agent. Boudreaux 
required about 5 hours to execute the 
plan autonomously. It eventually 
arrived at Pooh’s Corner, while the 
crew monitored its progress via 
Boudreaux’s video camera (Fig. 17). 
Excitement was highest when 
Boudreaux took its first panorama at 
waypoint 10 at the entrance of Pooh’s 
Corner. It took about 12 minutes to take 
the panorama and stitch it together into 
one image (Fig. 1 8). The picture was 
stored in ScienceOrganizer (Fig. 19) in 
the hab and then copied to the mirror 
site at Ames. Email was automatically 
sent to the RST. 



Figure 18. Boudreaux's first panorama at entrance of Pooh’s Corner (April 27, 2004). 



Figure 17. Kim Tyree monitoring Boudreaux’s traversal inside 
MDRS with video, mapping, and other telemetry from the ERA. 
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Figure 20. Crew’s analysis of Pooh’s Corner reconnaissance (excerpt) with next day EVA planning. 
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Boudreaux returned to MDRS after driving more than 1 km. The reconnaissance was successful, although we did 
not receive all four panoramas as planned because of a communication error on the third attempt, which propagated 
to prevent the fourth instruction from working. As in operating the MER rover on Mars, such problems can usually 
be resolved and the software updated during the mission. 

While Boudreaux was returning, the crew analyzed the panoramas and discussed a plan for the next day’s EVA 
to the same site. Figure 20 shows the Compendium map of that preliminary planning discussion. This information 
was sent to the RST using the Meeting Replay tool, with the expectation that the RST would annotate the plan. 

B. Day Two EVA to Pooh's Corner: Astronaut EVA 

Before the 2004 field season, we had made a detailed schedule of the two weeks. After the ERA’s 
reconnaissance of Pooh’s Corner, the astronauts were planned to return to the site with the robot the next day to 
perform geology exploration. However, severe winds forced the EVA to be rescheduled a day, which provided an 
additional day of lab-bench testing and refinement of the EVA plan. 

The weather April 29 was cold and cloudy, permitting the EVA to proceed. But when the ERA team turned on 
Boudreaux its motors were not working. An hour delay (until noon) was anticipated, however at 12:30 PM we 
decided that the astronauts should prepare, and we would do the EVA without the ERA if necessary. Then it began 
to rain. At 1 PM the rain had stopped and we started to suit up the astronauts. Preparing the new backpack with 
dGPS and two laptop computers, as well as getting two microphones fitted in the astronaut’s helmet (one for the 
speech dialogue system and one for the crew voice loop) and starting the Mobile Agents system required an hour. 
Coincidentally, HabCom reported that the plan was now ready in the MA system (Fig. 21) and at the same time the 
ERA team reported that the robot was ready, too. At 2:00 PM AstroOne (Brent Garry) said to his personal agent, 
“Start Egress activity.” 



Figure 21. Day two astronaut EVA plan for Pooh’s Corner. Each icon represents an activity object in the MA 
system, with location and timing information used by agents for alerts, as well as for automatically associating data 
in ScienceOrganizer. 

All systems go, AstroOne then said to his personal agent, “Start walk to waypoint 12 activity.” After several 
seconds AstroOne’s PA dialog agent said in a new female voice, “Walk to waypoint 12 activity started” (refer to 
Fig. 5). Next, AstroOne said, “Boudreaux, follow me.” Boudreaux’s PA received the command a couple of seconds 
later and forwarded the command to the ERA’s Communication Agent (a Brahms agent written in Java), which 
formulated and sent appropriate instructions to the robot control software via the ERA’s API. Boudreaux locked in 
on AstroOne’s GPS from his backpack and started following the astronaut at a safe distance. The EVA had begun. 
Everything was working, and the astronauts proceeded on foot to waypoint 12 at Pooh’s Corner. It had taken 
Boudreaux almost four hours to get there on Tuesday during its autonomous traverse. However, today it was 
tracking the astronaut in a level area, and so we were able to turn off automated obstacle avoidance. The astronauts 
reached waypoint 12 in 30 minutes with Boudreaux following. This 700 m traverse was a record for Boudreaux 
while following a person. 

A potential problem at the beginning of the EVA now became known to AstroOne. Early on, we learned in the 
hab while monitoring all of the agents from the HabCom computer that AstroOne’s GPS agent in the Brahms model 
was not receiving any GPS data from the GPS system. HabCom inferred that AstroOne’s PA would probably not be 
able to download his voice notes. This was confirmed when AstroOne finished his first voice note, and HabCom 
detected that the “speech act” from AstroOne’s PA did not arrive at the HabCom PA running on the HabCom 
computer. Thus, none of AstroOne’s voice notes could be stored inside ScienceOrganizer or Compendium. 
However, AstroOne’s dialogue system was recording his voice notes and storing them on the backpack computer, so 
they could be retrieved after the EVA. This combination of backup systems and debugging telemetry was an 
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important part of the MAA design for the 2004 field season, based on difficulties observed in previous years. Put 
another way, we were now using the MAA architecture for runtime monitoring, diagnosis, and repair of system 
components during EVAs themselves. 

A second problem occurred when AstroTwo attempted to download all images. HabCom detected, again inside 
the hab using the remote monitoring tools, that AstroTwo’s camera had inexplicably been associated with drive E: 
instead of the normal drive D:, and thus her PA could not find the image directory. HabCom was able to remotely 
change how the camera was on AstroTwo’s backpack computer. She repeated the command, “Download all 
images,” and then less than a minute later HabCom detected the pictures being downloaded into ScienceOrganizer in 
the hab. 

But then the process responsible for communicating with ScienceOrganizer inexplicably quit. This meant that 
the RST would not receive any of the science data during the EVA itself. HabCom detected that the RST was online 
and examining the problem remotely. The problem was resolved half way through the EVA; data did enter into the 
ScienceOrganizer database, and the RST received corresponding email (see Fig. 22). 
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Figure 22. Science data for Pooh’s Corner EVA recorded in ScienceOrganizer. Data is listed in hierarchical 
index on the left; information about selected entry is provided on the right. 


After almost three hours, the batteries of the backpack computers were almost empty. The astronauts’ PAs 
monitor the battery level and warn the astronauts and HabCom of low battery levels. We hot-swapped the batteries, 
providing another two to three hours. About 5:00 PM, approximately three hours since egress, we decided that the 
crew should start returning to the hab (the EVA was planned, but timing was flexible to allow for appropriate 
diagnostics and testing of the system, with the total time determined mainly by the batteries). The EVA crew had 
completed three of the four sampling activities on the plan and asked Boudreaux to take a panorama and a number 
of images. AstroOne walked over to Boudreaux, stood in front of his cameras and spoke again, “Boudreaux, follow 
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me.” Like a trusted mule, it pulled the science trailer and followed the astronauts back to the hab. The ingress 
activity started at around 5:15 PM. The EVA times and distances appear in Table 1. 


Table 1. Distances and duration outside hab during Pooh’s 
Corner EVA. 


But work was not yet finished for the 
astronauts. After removing suits and a 
debriefing with the MA team, the geologists 
started sample analysis in the lower floor of the 
habitat. The RST had provided a sample 
methodology process that included taking 
photographs of the samples inside the hab and 
providing additional voice notes. The MA 
system was shut down in the backpacks, but the 
HabCom computer stayed operational to allow 
the astronauts to create these voice notes and to 
download pictures from their cameras, so that 
they could be stored and associated 
automatically with other data from the EVA. At 
least this was the plan. 

Unfortunately, when the HabCom computer was brought from the upper deck, the system was unable to recover 
from a broken state. The speech dialogue system became unresponsive, and the crew could add no more science 
data. After dinner, we decided to use Compendium instead. Both geologists used their stored and downloaded 
images and voice notes to provide an analysis of the EVA. Finally, by midnight we were able to export the crew’s 
EVA analysis from Compendium and upload it into ScienceOrganizer, just in time for the RST to download it and 
have their 5 AM SOWG meeting to provide feedback to the crew. 


Robot/Person 

Distance 

Traveled 

(m) 

Duration 

(hr) 

Boudreaux 

1017.2 

2.748 

AstroOne (Brent) 

2745.45 

2.757 

AstroTwo (Abby) 

1983.01 

2.745 


C. Geologists’ Report on Pooh’s Corner EVA**”* 

Overall impression: Extremely successful EVA in comparison to last field season in regards to the robustness 
and sophistication of the system. The EVA lasted 4 hours in a very “real-time” and realistic atmosphere. The ERA 
played a prominent and crucial role in helping the astronauts achieve their field goals by providing a workstation 
and reducing fatigue by carrying important equipment. Our accomplishments are summarized here. 

1. EVA Plan 

We accomplished all tasks up through “Sample at Red Hill.” Time constraints prevented “Sample at Little Red 
Hill,” “Soil Core” activities, and “Stow equipment.” “Walk back to Hab” was manually instigated by HabCom. 

2. ERA and Science Trailer 

• The ERA followed AstroOne to and from the field with no manual control, using the “Boudreaux, 
Follow Me” command. 

• The ERA served as a base of operations for sampling at Rock Hill and Back Hill locations. 

• Science Trailer is useful as a storage tool for samples and equipment. There were six sample bags 
between the two astronauts. Samples and tools would have been too cumbersome to transfer between locations in 
the field and to carry back to the hab at the end of the EVA. 

• Improvised booklet of maps, panoramas, and voice commands was attached to the top of the Science 
Trailer as a field reference. This proved useful when we needed to refer back to new voice commands. 

5. AstroOne 

• My first EVA with full human-robot interaction with the robot acting autonomously to my commands. 

• Majority of time was spent working with commanding Boudreaux (the ERA) such as follow me, take a 
panorama, take picture of me, and watch me. 

• Was able to sample at Rock Hill and The Mount using the nifty nabber, special tongs to grab the rock 
samples without bending over. 

• Able to create sample bags with the Letter Letter / Digit Digit / Digit Digit format. With this format we 
could indicate sample location, astronaut who took sample, and sample number (e.g. RK/11/01 is for RocK hill 
taken by AstroOne at station 1 sample 01). 

• Able to lead ERA to the field and return the robot safely. 




This is a lightly edited first-person account written by the two geologists immediately after the simulated EVA. 


25 



4. AstroTwo 

• Managed to follow the majority of the plan (e.g. Egress, Starting Bio-sensors and GPS tracking, and 
starting Sample at Rock Hill). 

• Able to name locations and associate images, voice notes, and samples with locations. 

• Sample bags could be associated with voice notes, however I had problems with associating images and 
voice notes. 

• Sample bags were taken at Rock Hill, Back Hill, and Red Hill with associated voice notes at each 
location and images were associated with each location. 

• Tried out a number of voice commands: What time is it? Where am I? How much time is left?, What is 
current activity? Responses were timely and accurate. 

5. Room for Improvement 

• Astronauts need to stay within safe and helpful distance to each other. This was exhibited by the need to 
help with stowing equipment, also during equipment failures (e.g. AstroOne’s camera batteries died [probably from 
the cold], and needed AstroTwo to take a photo, while at Back Hill). The other astronaut serves as a useful scale in 
photos and to indicate cardinal directions. 

• Push-to-talk (PTT) function on the radios limited the astronauts in discussing what tasks to perform and 
observations and interpretations of the outcrops. A VOX communication system would be much more helpful and 
natural for the astronauts to communicate with each other. The PTT function interrupts making voice notes and is 
mentally distracting from the task at hand. Finding the button or waiting for someone else to finish talking is time 
consuming. Communication about what tasks the other astronaut has accomplished at the outcrop is hindered by 
different styles of working and lack of delegation of who was to perform which tasks. 

6. Post-EVA Analysis 

• Samples were photographed individually in the Hab. The astronauts used compendium to create a map of 
their own descriptions, observations, and interpretations for each sample and location. This was sent to the RST to 
help them with their analysis. 

VIII. Head of Canyon EVA Scenario Planning 

The second week of the field experiments was devoted to another site, about 5 km from MDRS. Three segments 
were planned for different days, at the head of the canyon (in which the ERA would provide surveillance by moving 
on voice command between predetermined waypoints at the canyon rim), around the bend, and down the long 
winding portion with the ERA (described in the next section). 

This section discusses the EVA planning by the crew of segment one, at the head of Lith Canyon. Two 
planning days were involved (May 3 and 4). This report demonstrates the workflow process and collaboration 
between the crew and the RST over two days. 

A. Crew Initial Planning of Head of Canyon EVA 

In the afternoon of May 3rd, the crew had an hour-long initial EVA planning meeting. In this meeting the crew 
had two objectives: 1) to review the available data for Lith Canyon and based on that define a doable EVA plan for 
next day’s EVA, and 2) to provide the RST with as good a background picture of Lith Canyon and the EVA plans in 
the coming three days as possible. In order to support the second objective, the crew decided again to record the 
meeting on video, so the Meeting Replay tool could be used. The crew started the meeting with the creation of an 
aerial map with and explanation of Lith Canyon (See Fig. 23). 

Using this map, the crew explained to the RST how they divided Lith Canyon into three segments, and how they 
planned two EVAs to geologically investigate the three segments. The next day’s EVA would cover Segment 1, 
while the EVA after that would cover both Segment 2 and 3 in one EVA. The scale of the aerial image in Figure 1 is 
1 meter per pixel making Lith Canyon about 600 m long. 

After the introduction of Lith Canyon to the RST, the crew focused on what they would be doing in next day’s 
EVA. To do this, the crew had created a hand-drawn map of Segment 1 , which included hand-drawn GPS waypoints 
for Boudreaux and for the crew to use for demarking the plan. (Fig. 24). 
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Sunlight direction 
from South East 


Segment 3 


Image Scale: 8 
800 x 600m I 


Segment 2 


Lith Canyon Local Area 


Segment 1 


The crew’s plan was to 
investigate Segment 1 by 
entering the canyon from 
Fossil Hill (waypoint 5). 
During the investigation of 
Segment 1, Boudreaux is 
to be commanded by 
AstroOne to move to 
waypoints 1 through 3. 
The goals were to provide: 
a) wireless communication 
relay back to HabCom 
while at the bottom of the 
canyon, and b) video feed 
and images of the 
astronauts in the canyon. 
To allow for total 
coverage in the canyon, 
the crew devised three 
waypoints at the rim of the 
head of the canyon, 
waypoint 1 allowed 
Boudreaux to cover the 
astronauts while they are 
descending into the 

canyon, as well as when they are working on the right side of the canyon (see Fig. 23). Waypoint 2 allowed 
Boudreaux to cover the astronauts when they are working on the left side of the canyon. Waypoint 3 allowed 
Boudreaux to visually track the astronauts when they are in the center of the canyon and moving towards the end of 
Segment 1 (referred to as “the Ledge”). The astronauts were able to command Boudreaux to move to either of these 
waypoints at will. 


Figure 23. Crew annotation of Lith Canyon aerial map. 



Lith Canyon with waypoints. 


The crew ended the meeting discussing how to 
proceed during the Segment 1 EVA. Some of this was 
captured in the Compendium map shown in Fig. 25, 
although the majority of the discussion was done using 
the maps of Fig. 23 and 24. It was therefore important 
for the RST to be able to view the Meeting Replay. 
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WayPoint4 


0 

Start Boudreaux Watch me when 
descending into the canyon starting at 
Fossil Hill 

0 

j can have Boudreaux take a picture 
ofAstroOne at anytime atterthat 

0 

Make sure that Boudreaux is in line of 
sight from AstroOne. ThU3 move it to 
WP 2 and 3 at appropriate times 

Figure 25. Crew’s initial planning map of Head of Canyon EVA using 
Compendium (excerpt). 



B. Crew RST Debrief Review and Final Planning of Head of Canyon EVA 

As planned, the morning of May 4 was used to set up the communication network to Lith Canyon. During this 
time the crew developed the final EVA plan. One of the first things noticed was that the crew downlink lead had 
made a mistake in the filenames of the downlinked video files the night before (the RST notified the crew of the 
problem by email). Consequently, the RST was unable to use the Meeting Replay tool during their early-morning 
meeting. This simple mistake was the result of fatigue, and illustrates why such tasks, which are easily automated, 
should be handled by personal agents. Here we were beginning to see how the MAA framework was useful for 
“office work” in the hab. This example illustrates how the empirically grounded development and field-test 
methodology helps determine requirements from a human-centered perspective. 

The RST was able to use the Meeting Replay tool later in the day and reviewed the video of the crew’s initial 
planning session (Fig. 26). 

Meanwhile, the RST had viewed the EVA plan (Fig. 25) in ScienceOrganizer, enabling them to provide some 
timely feedback to the crew for its final EVA planning session on May 4 (Fig. 27). The crew incorporated all of the 
RST’s suggestions in the final EVA plan. 

Weather prevented performing the EVA on May 4 as planned. The RST used the Meeting Replay tool and 
provided further feedback to the crew at the end of the day. The EVA was then run on May 5. 
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Figure 26. Meeting Replay of Crew’s Head of Canyon 

EVA Planning. Video inset shows the crew inside MDRS 
using Compendium and reviewing the aerial and hand-drawn 
maps to plan the EVA. 


On the way into the canyon, AstroTwo made a 
voice note at Fossil Hill, while AstroOne made a 
panorama from canyon floor. ERA successfully 
watched AstroOne for most of the EVA and took 
an image of him. AstroOne worked on the South 
Cliff Face, whilst AstroTwo worked on the East 
Cliff Face, taking voice notes and samples, with 
images. The EVA was ceased due to lack of 
sunlight about 8:30 PM. 


I TXi 

rstlithsegl crewplandebrief.txt 


RST Suggestions? 


<D 


1. Change Where is Pooh's Cornerto 
Where is Lithe Canyon (technicallity) 


(D 


,2. Have ERA take panorama at 
Waypoint 0 


'3. Have ERA take photo of Astro One at 
Waypoint 6 


<d: 


4. Have ERA take photo of Astro One 
when Astro One takes pan images that 
include locations that will 


Figure 27. RST email feedback brought into Compendium by the crew. 


The following section provides the Commander’s view of the subsequent day, preparing for the Segment 3 EVA on 
May 7. 


IX. Preparations for Lith Canyon EVA §§§§§ 

Log Book for May 6, 2004 
Commander's Check-In 

Time: started 8:19 AM May 7, completed 9:51 AM 

Weather: low 15c (59F) high of 29.1c (84.4F), hot and dry (12% humidity) with mid-afternoon winds 
Crew Physical Status: Very good, one person with minor scrapes. 

Narrative of Field Mission Results 

Today, May 6, was another successful day. We began with a briefing at 9 AM, joined by a NASA Ames 
management observer. 


§§§§§ -phis is an edited version of the report submitted by the expedition leader. 
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Preparing for the day’s test required a number of steps, which would take most of the day: 

1. To deal with the overloading of the ERA’S computers, we would use the spare laptop for running the 
Brahms-ERA agent system, locating it in the rental truck at Lith. This computer would be virtually “on board,” 
communicating wirelessly with ERA systems. Configuring the machine and relating it properly to the ERA might 
seem complicated, but the modularity of our systems made it relatively easy. 

2. To handle the overheating of the astronaut computers, we would reconfigure the ventilation in the metal- 
enclosed backpacks, adding small fans right against the CPU area, with an additional opening on one of the 
backpacks. We would use a flow-through method, rather than just pushing or pulling the air. The metal cutting and 
electrical work required a good part of the day. 

3. Once the first two steps were completed, we’d do a full-up start procedure in place at Lith, without 
intending to run a simulated EVA. We would verify the sequence of starting we had identified yesterday (roughly: 
network, RIALIST, Mobile Agents, biosensors, and GPS). We've learned that the sequence of plugging in the 
headsets, sensors, and camera is crucial; unplugging after the system starts can cause odd behavior later. 

4. We would examine the far point Cisco repeater (.5 km from our EVA site) and recharge the battery if 
necessary. 

5. We’d replace the push-to-talk mechanism used by AstroOne. 

I retired to my stateroom-office to write the previous day’s report and process photographs. I also greeted a 
Spanish-speaking TV group from Salt Lake City and gave an orientation to the Ames visitor. These activities filled 
my morning, then I turned to email, including responding to more than 20 fairly interesting questions from an Italian 
reporter. 

The winds picked up at the canyon, requiring equipment to be moved into the truck and the canopy to be tied 
down even more securely. But the network backbone held with good throughput to the hab. We decided to go out 
about 4 PM after testing showed that the fans allowed the backpack computers to run cooler than those just sitting 
on tables in the open truck. An ERA team member had carried around the backpack doing speech tests with the 
ERA, which showed alj was in order. We decided to do a short EVA, rerunning the head of canyon scenario. 

What followed was our quickest start to date: We arrived at Lith at 4:40 PM, started the Mobile Agents system 
from 5-5:13 PM, changed to internal battery at 5:18 PM. By 5:30 PM AstroTwo was fully outfitted and starting her 
sensor systems by voice command. The EVA then proceeded in its usual chaotic way, lasting nearly 1 .5 hours. 

Conditions were excellent for this EVA test — moderate to no wind, temperatures in low 80s, partly cloudy, 
beautiful late afternoon lighting. Of course not everything went according to plan. 

The ERA had some GPS trouble and did not ever reach a waypoint by voice command (though it came close on 
the last try, losing GPS during the operation). The robot was successfully teleoperated from the hab to provide video 
of the work in the canyon. The astronauts commanded the robot to take photos and a panorama from the ledge, 
looking down at them on the canyon floor. (However, one photo was a bit late, and AstroTwo had already moved 
out of sight.) 

For about an hour, the geologists went about their work of naming work locations, taking photographs and 
samples, and making voice notes. New commands like “Associate last voice note with last sample bag” and 
automatic naming of downloaded images as “collections” occurred properly. By using location names that begin 
“Work Site” (e.g., WorkSite5), the system automatically associated subsequent sample bags and voice notes with 
that location (so additional commands are no longer required). The data were properly stored in the 
ScienceOrganizer database in the hab (transferred automatically by Mobile Agents from the HabCom Brahms 
system) and copied by MySQL to the mirrored database at NASA Ames. 

Members of the RST received email of each filing in ScienceOrganizer (similar to Fig. 8). Notice how the EVA 
plan, activity (established by the astronaut during each phase of the EVA), astronaut name, exact location and time 
are stored. 

As a member of the hab crew, I received copies of each message, so I can quickly review the sequence of what 
occurred during the EVA from my mail folder. I see that AstroTwo took 19 photos, used four sample bags, and 
made three voice notes. She named three work sites, spaced 20 and 16 minutes apart. An email even indicates the 
moment when her GPS signal was lost (as she stepped under an overhang at the lip of the canyon) — a milestone for 
this test, because her backpack computer remained on the network, with the ERA serving as a relay to the backbone. 
Various alerts about heart rate and Oxygen saturation were also sent, but it’s not clear whether these sensor readings 
were correct. 

Although we were disappointed that the ERA could not be moved from side to side by voice command to keep 
the astronauts within camera view, this was actually an improvised addition to the scenario last week. We in fact met 
our objectives for this segment of establishing the ERA as a relay and using the Mobile Agents system to properly 
document the geology of an area. 
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In comparison, last year’s test at Lith lasted only about 20 minutes and only proved that the basic data logging 
commands worked.; only one image was actually received remotely. This year we used the system for over an hour 
in the canyon, used much more sophisticated naming and association automation, and all of the data were stored 
automatically in the hab and a remote database. 

We have clearly turned the corner from just getting the system to work at all, to having a system that can actually 
be used experimentally and improved by observing how people use it. 

We arrived back in the hab before 8 PM. We were treated to a pizza dinner by NREN, including the entire 
expedition team (with visitors from two communication network vendors). It was a fun and relaxing evening. Our 
celebration was tempered only by the absence of the support person who had set up the party, now hospitalized in 
Provo with a heart problem (though doing well). 

The intensity and excitement of the day’s work continued late into the evening, as we watched a “video file” on 
NASA TV after 1 1 PM. The 17 minute segment, filmed and edited by a team at NASA Ames, showed our work last 
week and included interviews with the crew. It was eerie to watch this video sitting in bed in my stateroom, using 
my laptop on the wireless network. Others watched on the big screen in the main room of the upper deck. 

With the draw on the network, the image was out of synch with our voices. And bizarrely, adding to the sense of 
relativity, my stateroom connection was about 20 seconds ahead of the wardroom’s, so I provided narration about 
what was coming next. Our experience was a hall of mirrors, of present and future echoing and twined together-just 
like our field tests, both physically real here and now, yet in our imaginations and videos, projecting the reality of 
people on Mars. 

Plan for Tomorrow: Do Segment 3, in which the ERA follows the astronauts in the long run out of the canyon 
towards the east and north. 


X. Long Relay EVA Scenario (Lith Canyon) 

This section provides reports from the ERA team and the geologist-astronauts on the May 7 EVA into Lith 
Canyon (Fig. 23). 

A. ERA Team Report****** 

May 7 was our one and only attempt for Segment 3 — an endurance challenge designed to run-out 
communications, data, batteries, and Boudreaux. 

Segment 3 required the ERA team to belay Boudreaux down to the bottom of Lith Canyon. There Boudreaux 
towed the science trailer carrying tools and equipment to assist the astronauts as they stressed the limits of the 
wireless data network, the mobile agents infrastructure, and their own stamina in the suits and the heat. 

• The ERA team left the motel at 7:30 AM to give plenty of time to belay Boudreaux the robot and its science 
trailer approximately 75 m along a sloped path down into Lith Canyon. Five men were required to perform 
this operation. 

• The EVA simulation started at approximately 20:10 UTC (2:10 PM local time). 

• The obstacle avoidance system came very close to failing once at 22:25 UTC, slightly scraping a large rock 
after two forward-backward-turn attempts around it. An e-stop was pressed by field support crew, but was 
strictly unnecessary. AstroOne was asked to issue a “Halt” voice command to reinitialize the rover (which 
was slightly odd from a human perspective, given that the rover was obviously stopped.) 

• ERA’s computer inside AstroOne’s infoPak backpack reached a peak CPU temperature of 44 C (1 1 1 F) and 
never failed due to heat. The MA computer inside the same infoPak did fail due to heat (something above 55 
C (131 F). CPU loading on the MA computer and poor venting at the laptop caused the high heat and failure 
(ambient temperature was 83 F with 7% humidity and no wind). 

• The simulated EVA ended at approximately 23:28 UTC. 

• Boudreaux suffered no hardware failures or anomalies today, and required no primary battery swaps. The 
ERA laptops had their batteries replaced at the start of the EVA start (they were up and running since 17:00 
UTC, when the EVA was originally scheduled to begin). After the run was over, one of the main laptops 
onboard Boudreaux failed (dead batteries) because it was not immediately removed and placed on AC power. 

• The following objectives were accomplished today from the ERA’s perspective: 

• Six “take image” commands 

• Four “take panorama” commands 

• Six “follow me” commands 


****** 


This is an edited version of the report submitted by the ERA team after the May 7 EVA. 
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• Five “watch me” commands 

• The interesting combination of watching one astronaut with a pan-tilt camera while following the other 
astronaut. 

• Four “stop” commands 

• Multiple “report position” commands 

• Multiple “define position” commands 

• One “report ground-track map” command 

• One “report elevation map” command (after a fixed bug on a failed first attempt). The map itself was not 
very useful since AstroTwo was not reporting elevation due to using only absolute GPS hardware. 

• Speech output from Boudreaux through Brahms out to the astronauts and HabCom (using new server and 
listener software). 

• Situational awareness of the remote EVA from HabCom inside MDRS was achieved by providing live 
imagery from Boudreaux and various GUIs, some showing the agent positions and their movements. 

• Boudreaux served as a data communications relay from the astronauts up to the repeaters (ATV and 
antenna on ridge, see Fig. 2) and back to the Hab. 

• A series of still images (1 fps) were collected of the entire EVA to be stitched together as a short movie. 

• Only one objective was not accomplished today: the “print curation label” command failed due to a hardware 
failure in the printer itself (probably due to sand in the mechanism). 

• The total times and distances traveled today were: 

• Boudreaux, 3.3006 hours, 289.937 meters 

• AstroOne, 3.2945 hours, 1740.931 meters 

• AstroTwo, 3.2882 hours, 1517.784 meters 

• After the run, five men again belayed the robot and science trailer up from the canyon floor and helped it get 
back to the camp. 

• The ERA team packed up and secured everything for the ride along rough BLM roads from the Lith site back to 
the MDRS Habitat and the motel for packing. 

To summarize the day, all field teams had an excellent run, and the ERA team performed well. The Lith Canyon 
challenge has finally been met, though not conquered. 

B. Geologists’ Report on Lith Canyon EVA tttt+t 

Lith Canyon had been separated into three segments, planned for three sequential days (Fig. 23). Due to numerous 
repetitions of Segment 1 to get all of the equipment functioning. Segment 2 was skipped in preference of Segment 3, 
which incorporated the ERA and gave more opportunity for geology. Segment 3 is summarized here. 

• Morning was spent creating a plan for Segment 3. This was difficult due to lack of images for this area - 
we decided to employ a basic “walk to the canyon, move along the segment and create worksites at areas 
we found interesting geologically.” 

• The astronauts were on the canyon floor by 2:20 PM (local MST) and the EVA ceased at 6: 10 PM - very 
successful EVA in all! 

• Creation of worksites and the subsequent association of voice notes, samples and images all worked well. 

• ERA followed AstroOne for the majority of the Canyon. 

• Astronauts were able to lead the ERA trough a complicated “S-turn” in the canyon. 

• ERA’s panoramas and images were commanded by AstroOne primarily with some by AstroTwo to 
demonstrate that either astronaut could command the robot. 

• Astronauts managed a good EVA with one battery change and a refill of water. Models only failed at the 
very end of EVA; AstroTwo’s model was shut down accidentally by HabCom [which indicated an 
opportunity for intervention by his personal agent!]. 

• Numerous worksites, samples, images and voice notes were created throughout the EVA. 

• Geologically, the astronaut’s walked “down-section” into older layers of rock than were found at the head 
of the Canyon. Exposure of fresh faces were more abundant in Segment 3 . 

• We were very tired at the end! 


tttttt 


This is an edited version of the report submitted by the geologists after the Lith Canyon EVA of May 7. 
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XI. Results and Discussion 

The Mobile Agents 2004 experiments demonstrated that the system has transitioned from an engineering test 
phase to experimental use doing authentic work, in particular geologic exploration in terrain that was new for the 
participating scientists. Different teams analyzed the data appropriately for their needs, focusing on performance of 
the robot (ERA), voice commanding (RIALIST), communications network (MEX), agent processing (Brahms), and 
work system design (i.e., correctness, efficiency, and naturalness of the overall tool-using experience relative to the 
work to be done in the canyon and habitat setting). 

Analyses are primarily statistical for each category, using history data recorded during the various scenarios. For 
example, a complete log exists for all Brahms agent state changes, including changes in belief, activities, and 
primitive actions (including communications between agents). This log may be viewed graphically, such that one 
may inspect at any moment what each agent on each platform was doing and how they were interacting. Similar 
information exists for the ERA, RIALIST, and MEX. An excerpt from the Lith Canyon (May 7) EVA is analyzed in 
the next section. The subsection following details many improvements based on the 2004 experiments. 

A. Example analysis of voice commanding interaction 

Of particular value is the complete record of speech from the astronauts and HabCom, as well as a record of 
what speech was generated by agents as questions or feedback. This data can be consolidated into a table, allowing 
the full chronology of interactions to be grasped (Table 2). The time is local MST. The astronaut designates one of 
the EVA crew members or HabCom. The origin indicates either a voice recording by an astronaut (.wav file) or the 
personal agent of an astronaut or the ERA (in some cases, it is actually the corresponding dialog agent that generates 
the speech act). Finally, the actual speech spoken by the astronauts or generated by the agent is given. When no 
speech appears, the astronaut’s remark was not interpreted to be a voice command (e.g., it might be a conversation 
with the other astronaut or an external support person). 

Several interactions are noteworthy, illustrating how the analysis process proceeds: 

o It can be seen that Boudreaux required nearly 1.5 minutes to halt. On observing this, and anticipating 
the safety problems it might cause, we have decided that confirmation of this command should occur 
after it is executed, rather than before. 

o Notice that when Boudreaux halts, all members of the crew are notified that the “tracking sequence is 
aborted.” The synthetic voice indicates that it is “spoken” by the ERA. This might be distracting to 
AstroTwo, who has just begun recording a voice note, and is not directly working with the ERA at this 
time. Hence, context might be taken into account in determining who receives this feedback 
information. 

o AstroOne’s sequential statements at 39:32 and 39:41 suggest that possibly uttl05.wav was a command 
that was not understood. We can listen to the file to determine the cause of the problem, and indeed use 
the recording to refine the speech recognition system. 

o AstroOne asks the ERA to take a panorama before the stop acknowledgment has been received. Before 
this command is processed, PA 1 tells the astronaut that it is still waiting for a response. But it may not 
be clear which command it is referring to at this point. It would have been clearer if PA 1 said, “I am 
still waiting for a response on your halt command.” 

o Meanwhile, AstroTwo has been recording a voice note, which lasts for about 1.5 minutes and is stored 
in the file indicated at 23:40:41. This interaction, like the halt and panorama commands from AstroOne 
are being relayed back to the HabCom computer, with corresponding data stored in ScienceOrganizer. 
The agent confirms that a database link should be created that associates the voice note with the 
astronaut’s activity (“work at waypoint 16”) and the work site (number 7) she named previously, during 
this EVA. (However, the feedback at 23:41:01 indicates an error, in referring to voice note 8.) 

o A similar interaction follows, in creating a sample bag with a name improvised on the spot. AstroOne is 
told at this point that the ERA’S panorama to not yet complete. When actual problems develop here, the 
HabCom should have been notified with an alert. 

o Keep in mind that while all of this is occurring, agents are simultaneously tracking biosensors, location, 
and duration within the activity, generating appropriate alerts when thresholds are exceeded. 
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Table 2. Excerpt of voice interactions during May 7 Lith Canyon Scenario. 


TIME 

ASTRO 

ORIGIN 

SPEECH 

23:38:46 

1 

utt103.wav 

Boudreaux stop 

23:38:46 

1 

PA1 

Do you want Boudreaux to halt? 

23:38:51 

1 

utt104.wav 

yes 

23:38:51 

1 

PA1 

Asking Boudreaux to halt. 

23:38:53 

2 

utt408.wav 


23:39:01 

2 

utt409.wav 


23:39:08 

2 

utt410.wav 

take a voice note 

23:39:08 

2 

PA2 

Do you want to record a voice note? 

23:39:12 

2 

utt41 1 .wav 

yes 

23:39:12 

2 

PA2 

Please begin recording voice note now: 

23:39:16 

1 

PA ERA 

Tracking sequence aborted 

23:39:18 

habcom 

PA ERA 

Tracking sequence aborted 

23:39:18 

2 

PA ERA 

Tracking sequence aborted 

23:39:32 

1 

uttl 05.wav 


23:39:41 

1 

utt106.wav 

Boudreaux take panorama 

23:39:49 

1 

PA 1 

1 am still waiting for a response on your command 

23:40:15 

1 

utt107.wav 

Boudreaux take a panorama 

23:40:16 

1 

PA 1 

Boudreaux stopped. 

23:40:16 

1 

PA 1 

Shall 1 take a panoramic image named panoramic image 1 here? 

23:40:41 

2 

PA 2 

Finished recording voice note 

23:40:41 

2 

voice_note_ 

2004-4-7_23-39-8.wav 

23:40:44 

2 

PA 2 

Voice annotation is created 

Do you want to associate Astronaut Two Work At Way Pointl 6Work Site7 

23:40:44 

2 

PA 2 

with Astronaut Two Model VOICE ANNOTATION 7? 

23:40:48 

1 

uttl 08.wav 

yes 

23:40:48 

1 

PA 1 

Taking a panoramic image named panoramic image 1 . 

23:40:58 

2 

utt413.wav 

yes 

Attempting to associate Astronaut Two Work At Way Pointl 6Work Site7 with 

23:40:58 

2 

PA 2 

Astronaut Two Model VOICE ANNOTATION 7. 

Voice annotation voice note 8 is associated with location Astronaut Two 

23:41:01 

2 

PA 2 

Work At Way Pointl 6Work Site7. 

23:41:24 

2 

utt414.wav 

create sample bag echo lima slash two seven slash zero one 

23:41 :24 

2 

PA 2 

Do you want to create sample bag echo lima / 2 7 / 0 1 ? 

23:41:31 

2 

utt41 5.wav 

yes 

23:41:31 

2 

PA 2 

Attempting to create sample bag echo lima / 2 7 / 0 1 . 

23:41:34 

1 

PA 1 

1 am still waiting for a response on your command 

23:41 :34 

2 

PA 2 

Sample Bag is created 

Do you want to associate Astronaut Two Work At Way Pointl 6Work Site7 

23:41:34 

2 

PA 2 

with sample bag echo lima / 2 7 / 0 1 ? 

23:41:50 

2 

utt416.wav 

yes 

Attempting to associate Astronaut Two Work At Way Pointl 6 Work Site7 with 

23:41 :51 

2 

PA 2 

sample bag echo lima / 2 7 / 0 1 . 

Sample bag echo lima / 2 7 / 0 1 is associated with location Astronaut Two 

23:41:54 

2 

PA 2 

Work At Way Pointl 6Work Site7. 
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B. Summary of Discoveries and Tentative Design Recommendations 

This subsection summarizes by component category the most salient observations made from the 2004 
experiments and design changes that are being considered: 

1. ERA Use during EVA 

o Tele-operate ERA (using video and pancam) to allow the crew to remotely explore new region and 
define work sites (with dGPS locations), avoiding physical walkthrough prior to an EVA. 
o Clarify why using the ERA with a single shared trailer in an area such as Lith Canyon is superior to 
using individual ATVs for the astronauts with trailers and independent video, 
o Investigate ways to increase speed while using obstacle avoidance (a problem of gathering sufficient 
data); tune motor controllers; use more powerful motors; add remote kill capability, 
o Add an arm and hand to each ERA 
o Build 3D panoramas 

o Add voice commands to enable astronauts to obstacle avoidance on turn and off (tests showed that it is 
unnecessary when following astronauts in relatively flat, unobstructed terrain), 
o Train astronauts using voice commanding and handling the robots more extensively prior to the field 
experiments (this is usually a matter of timing and availability, as the astronauts are students in NY), 
o Add/Improve robot speech so Astronauts have better awareness of robot activity. Specific examples: 
ongoing updates on tracking, taking panoramas to reassure astronauts that robot is healthy, and obeying 
commands. 

o Make the ERA PA more proactive in monitoring safety and ease of use: 

• During Voice Note recording, ERA automatically takes photograph of the astronaut to establish 
the setting 

• ERA automatically follows astronauts and watches them when they are moving to the next 
activity (ask for confirmation) 

• “Halt” is executed first, then confirmed; continue moving otherwise 

• Monitor astronaut location with respect to communications relay (maintain line of sight with both 
astronauts and ensure that they can see each other) 

• Do image processing to detect an obscured astronaut or one in medical trouble (combining with 
biosensor data and activity expectations, e.g., when is lying prone on the ground acceptable?). 

• Reject move commands that will break communications relay 

• Detect and locate objects by gestures, e.g., “Take a picture of that rock.” 
o Use a second ERA robot and/or a Athena (MER-class) robot during EVAs: 

• Leave a robot behind to do analysis for reporting during the EVA, providing information useful 
when the astronauts return later to that spot. 

• Use a second robot as a communications relay, moving the two robots in a teamwork manner to 
accomplish better coverage 

• Teleoperate a robot from the hab (or via program uploads from “Earth”), using it for 
reconnaissance of potential EVA sites or revising places identified by the astronauts as worthy of 
detailed investigation. 

2. Agent Language Improvements 

These suggestions are based on interactions between the astronauts the support personnel, indicating 
communications that should be had with their personal agents: 

o Provide information about the robot’s state, e.g., “Boudreaux, whom are you following now?” 
o Provide information for navigation, scheduling, and work activities: 

• Where is {waypoint # | the next waypoint}? 

• Where is the next activity? 

• How do I get to location X? (X = name in the plan) 

• Where is astronaut N? 

• What is astronaut N doing? 

• When do I/we need to turn back? 

o Information about EVA histories, e.g., “Where did I/we sample X previously?” 
o Enable astronauts to classify samples using descriptive language 
We also need to properly implement “generic” activities based on EVA ontology: 

{Move, Sample/Survey/Document (photo + voice note). Apply Instrument} at <loO for <duration> with 
<astronauts, robots> 
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3. EVA Planning and Data Display 

These suggestions concern the collaboration tools used by the crew and the RST: 

o RSS-style (Really Simple Syndication) EVA data display during the EVA to which RST and support 
teams can subscribe, complementing or replacing emails. 

• Summary and highlights of the EVA to a given point 

• Resource statistics (time, distance), alerts, key URLS 

o EVA “replay” tool - synchronized playback of astronaut and agent voices (now recorded separately 
in different folders for each speaker), plus agent activities (workframe state), showing alerts, 
movements 

o Above to be integrated with Brahms Virtual Environment 40 , a graphic-animation of the EVA for 
viewing state and accessing science data. 

o Personal agent to help analyze EVA data in the hab (“CEV-style downlink agent”), including post- 
EVA curating 

o Better integration of ScienceOrganizer, Compendium, route planning and mapping software, with 
data analysis tools (e.g., Excel spreadsheets) 

o Develop a new planning tool with a customized planning user interface on top of Compendium to 
allow for more structured planning and simplified information entry and validation. The planning 
tool can be used during an EVA to visualize the status of the plan by interpreting events 
communicated by the agents. 

4. Summary of significant configuration requirements for the MAA 

o Improved plug-n-play of MA platforms, e.g., carry on an astronaut EVA without the ERA or 
dynamically add a robotic module during an EVA 

o Move plan management from HabCom agent to astronaut PAs, improving response time and safety 
in the event of a communications failure, plus allowing each person and robot to have an independent 
(or specialized) plan and to be able to be run independently of the HabCom computer, 
o Separate ScienceOrganizer Agent from PAs on each platform for efficiency. 

o Eliminate requirement for a second laptop running Linux in backpack for dGPS (by porting software 
to Windows); will reduce chance of overheating in the backpack, 
o Move laptop that manages agent registry from ATV serving as a relay to ERA trailer for more 
realistic configuration, including better directory service and an agent search function, 
o 400 MHz walkie-talkies for all participants to eliminate separate system for astronaut 
communications. 

o Enhanced system monitoring that includes CPU temperature monitoring to prevent unexpected 
computer failures. 

o Develop a procedural agent that relates astronaut actions to semantics of generic activities, to allow 
CapCom-style monitoring of what the astronauts are doing at each work site, e.g., alert if did not take 
a sample during a site survey activity. 

o Develop a medical agent for more systematic health monitoring and alerting of crew members during 
an EVA. 

o Develop a navigation agent that reorganizes functions currently in the HabCom PA and has improved 
capabilities to respond to information requests and commands (listed above), 
o Similarly, reorganize data storage and proxy agent functions for increased efficiency and modularity, 

o Store and make available EVA data to the agent system over long durations (starting with days and 

continuing to years). Reminding and alerting functions can be augmented to search and relate current 
activities to past data, methods, difficulties, etc. 

o Improve agent models overall to retain and work within activity contexts, rather than being strictly 
reactive (speech-act driven). 

We believe that the increasing number of ideas generated from field experiments each year indicates the rich 
potential of our approach, involving incremental development of a useful tool, within an authentic work context, 
with broad involvement of many roles in integrated simulations. The Conclusions section comments on how this 
work fits within the overall objectives of planetary EVAs and how the sequence of Mobile Agents experiments 
exemplifies an analog missions field campaign. 
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XII. Conclusions 

The Mobile Agents Architecture is now ready for development in space exploration missions to support a 
network of coordinated operations between astronauts and remote support teams, robotic systems, other devices and 
systems, and software tools. The key aspects of the architecture are: multi-layer communication network relating 
wireless services to agent software; distributed computers, robotic devices, sensors, and instruments; workflow 
agents (including monitoring, diagnosis, advising, and data management); a commanding and feedback protocol; 
semantic database with automated, context-based creation of associations; activity-based analysis and design; tools 
to facilitate remote collaboration. At the same time, the use of speech recognition, GPS, a wheeled robot, etc. are 
important in advancing the state of the art, but not essential elements of the computing methods and architecture. In 
practice, the ERA might be replaced by a personal ATV on which an astronaut could ride, perhaps with tracks rather 
than wheels; a touch screen on a spacesuit cuff could complement or replace speech processing; GPS might be 
replaced by any number of alternative location and navigation technologies. The essential idea is gluing together a 
diverse, distributed set of devices, software, and players by using a multi-agent system to monitor and manage 
workflow. 

Although we believe that the MAA system has reached a state of practical development, it should be 
emphasized that the vision of automating CapCom requires many more years of work. Agents that do health 
monitoring are only proof of concept, lacking most of the functionality required. The foundation is laid for 
navigation by allowing astronauts to name places, but nothing has been implemented for helping the astronauts 
identify where they are or how to get to other places. Many functions have not even been prototyped, such as 
replanning EVAs dynamically with teams of people and robots, and handling emergencies. Although the current 
voice commanding is impressively robust and useful, the dialog system has no capability for real conversational, 
mixed- initiative interaction, which the state of the art in computational linguistics allows. Within each of these 
arenas, points of flexibility need to be identified (as outlined in the first paragraph of this section), such that trade 
studies with competing approaches and technologies can be carried out. 

Referring to CapCom’s role in Apollo lunar traverses, one of the most demanding activities involves advising 
the astronauts about when to move along to the next site, when to take extra time (and what to do), and when to skip 
part of the EVA plan. CapCom provided this advice by interacting with the science backroom (in a room located 
near to the Mission Control Center). For extended lunar missions most of the scientists will be distributed in their 
home locations, as they are today in the extended MER mission. Furthermore, for Mars missions and beyond, the 
time delay will prevent sufficient awareness and time to advise work that will occur within 15 to 50 minutes 
roughly, allowing time to receive data, interpret it, and to reach a consensus opinion. Thus, as many people have 
noted, 41 EVA advising from Earth will be more strategic, considering steps in the current EVA and of course 
planning work for future days. The role of the PAs on the moon or Mars will accordingly require sending important 
information during EVAs to key personnel, relating current results to the objectives of the EVA (e.g., to detect when 
results are disappointing or not significant and allow for cutting time short), and revising the plan being followed 
inside the MA system according to information being received from remote teams during the EVA. One idea is that 
scientists on Earth could have personal agents that themselves “come alive” at appropriate moments during an EVA, 
serving as proxies that inform or remind astronauts of interests as opportunities as they occur (e.g., “AstroOne, that 
rock appears to contain goethite, remember that Dr. Z asked for an underlying soil sample when you found goethite 
...”). Developing such capabilities in a robust way approach some of the classic artificial intelligence problems of 
modeling new phenomena on the fly. 

In the convergence of new exploration missions (e.g., the Crew Exploration Vehicle) and maturing technologies 
such as Mobile Agents, concern increases that research activities should “flow down” from requirements. Although 
returns on investment are important in the long-run, delivering according to specifications is not how research and 
invention occur. In practice, scientific investigation — which is required to properly relate people and robots in 
extreme environments — develops from existing instruments, methods, scenarios, settings, and questions. This 
repertoire of tools and capabilities can be redirected to meet mission objectives, but always some freedom for 
exploration — here in the scientific domains of cognitive science, anthropology, and sociology — is just as necessary 
as it is for the geologists and biologists who investigate the universe. 

In practice, a “middle out” approach, rather than a strict “flow down” from requirements is possible and 
advantageous. Scientists already engaged in experimental work can become engaged in near-term mission needs 
(i.e., for use within 5 to 10 years) by finding connections between advanced approaches (such as Mobile Agents) 
and operations requirements. To a large degree, existing research provides essential operations concepts that would 
not be considered if one were only specifying systems according to historical metaphors and tools currently in use. 
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For example, consider a Mars scenario with a crew of four in a surface habitat, a MER-like robot regularly 
inspecting external systems, robots like the ERA, and EVAs for two crew members of varying durations and 
distances from the habitat. To what extent can the crew inside the habitat work unimpeded on other tasks, such as 
writing reports, analyzing data, planning future work, repairing equipment, etc.? If one starts with historical 
concepts based on the Space Shuttle, one would assume that one crew member will be dedicated to monitoring the 
EVA for its full duration, maintaining a continuous conversation with the crew and guiding them. But Mobile 
Agents results suggest that a variable degree of attention is possible, such that responsible crew member (whom we 
have called HabCom) receives status information that varies from continuous notification (e.g., new activities started 
by any person or robot) to emergency only alerting (e.g., an astronaut health problem, equipment malfunction, or a 
violation of a plan constraint that is not resolved by the crew within a certain time period). We know generally how 
to build such a system today using the MAA. The questions to be determined are: How safe and practical is 
emergency only alerting (e.g., do the astronauts need to consult with HabCom for many routine problems)? Given 
the loss of situation awareness by not paying attention, and the possible difficulties of reengaging in what is 
happening during the EVA, is there an optimal kind of notification that facilitates handling emergencies or even 
detecting and handling limitations in the agent system? To answer such questions, we could develop a Mars analog 
simulation (such as at MDRS), include an Earth-based safety monitor role, and experiment with different designs. 
This example shows that even such a basic question as whether a crew member must always attend to an EVA is 
called into question by new technology. The implications potentially and dramatically change how crew activities, 
monitoring tools, and software throughout the system is conceived and built. 

In the near term, Mobile Agents experiments will shift from running the system until it fails or the astronauts are 
exhausted, to full circuit simulations that monitor resources and advise about turn around time and attempt to extend 
explored areas through EVAs on different days. Thus, schedule priorities, teamwork, and well-timed coordination 
with the RST will become important. In this way, the system will incrementally become more and more like 
Apollo’s CapCom in its capabilities. Following a human-centered approach, we will continue to emphasize how 
people and our computer systems differ, 15,16,42 thus highlighting what tasks must be monitored or carried out by the 
astronauts or HabCom. We believe that this is the only way to reliably, realistically, and efficiently produce human- 
robotic systems— developing on a foundation of what is useful, remaining cognizant of what is possible using the 
technology, and yet have our eye on a vision that stretches technical capability and extends the reach of space 
exploration. 
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